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Abstract 

Background: In assessing quality of care in developing countries, retrospectively collected data are usually used 
given their availability. Retrospective data however suffer from such biases as recall bias and non-response bias. 
Comparing results obtained using prospectively and retrospectively collected data will help validate the use of the 
easily available retrospective data in assessing quality of care in past and future studies. 

Methods: Prospective and retrospective datasets were obtained from a cluster randomized trial of a multifaceted 
intervention aimed at improving paediatric inpatient care conducted in eight rural Kenyan district hospitals by 
improving management of children admitted with pneumonia, malaria and diarrhea and/or dehydration. Four 
hospitals received a full intervention and four a partial intervention. Data were collected through 3 two weeks 
surveys conducted at baseline, after 6 and 18 months. Retrospective data was sampled from paediatric medical 
records of patients discharged in the preceding six months of the survey while prospective data was collected from 
patients discharged during the two week period of each survey. Risk Differences during post-intervention period 
of 16 quality of care indicators were analyzed separately for prospective and retrospective datasets and later plotted 
side by side for comparison. 

Results: For the prospective data there was strong evidence of an intervention effect for 8 of the indicators and 
weaker evidence of an effect for one indicator, with magnitude of effect sizes varying from 23% to 60% difference. 
For the retrospective data, 10 process (these include the 8 indicators found to be statistically significant in 
prospective data analysis) indicators had statistically significant differences with magnitude of effects varying from 
10% to 42%. The bar-graph comparing results from the prospective and retrospective datasets showed similarity in 
terms of magnitude of effects and statistical significance for all except two indicators. 

Conclusion: Multifaceted interventions can help improve adoption of clinical guidelines and hence improve the 
quality of care. The similar inference reached after analyses based on prospective assessment of case management 
is a useful finding as it supports the utility of work based on examination of retrospectively assembled case records 
allowing longer time periods to be studied while constraining costs. 
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Background 

Effective organization and provision of care for children 
admitted to hospital with acute illness in developing coun- 
tries is advocated through World Health Organization 
and Kenyan guidance [1-4]. The effectiveness and cost- 
effectiveness of one approach to their implementation in 
Kenya is reported [5]. As with other studies evaluation, 
the study was based on retrospective examination of case 
records [6,7] that are typically more feasible than examin- 
ing inpatient care prospectively by observation. The ap- 
propriateness of using medical records in epidemiological 
research depends on the accuracy of transfer of informa- 
tion from patient to clinician, the accuracy and level of de- 
tail in recording that information onto medical records by 
the clinician and how well abstraction of information from 
medical records is done [8]. The quality of data obtained 
from medical records can be compromised by unclearly 
specified research questions, vague specification of vari- 
ables, poorly designed abstraction tools, poor understand- 
ing of the data by the abstractors and incompleteness 
of the data in medical records [9]. Comparison of self- 
reported data with data abstracted from health records on 
initial treatment of prostate cancer showed good agree- 
ment between the two datasets [10] while comparison of 
prospectively collected data with retrospectively collected 
data in studying risk factors for coronary artery disease 
showed that medical records are less accurate and less 
complete compared to prospectively collected data [11]. 
Eder et al. discusses some of the strategies that can be 
used in enhancing the reliability of data abstracted for 
medical records. These include establishment of priority 
in the source of information if there are multiple sources 
of information and having standardized terminology [12]. 
Abstracting data from medical records can be preferable 
to prospective designs because they are less resource in- 
tensive, they can be easily used in exploring possible asso- 
ciations and can be performed at researchers' convenience. 
However, such data are subject to numerous sources of 
bias, usually have a lot of missing data especially where 
documentation of care is poor and it's often difficult to es- 
tablish true causal effect relationships [8]. Here we use 
prospectively collected case data to examine the effects of 
an intervention aimed at improving paediatric practices in 
Kenya and contrast these findings with those previously 
reported based on retrospectively collected data [6]. Our 
aims is both to examine the intervention effects using ar- 
guably a more robust data set and triangulate the findings 
with those based on retrospective case record review. 

Methods 

Study sites and participants 

Eight rural district hospitals from four of the eight prov- 
inces were purposively chosen to represent rural district 
hospitals in Kenya and have been described in full 



elsewhere [1]. Neither the Ministry Of Health nor the 
hospitals had any defined procedures for implementing 
new clinical guidelines prior to the study. The Kenya 
Medical Research Institute National Ethics and Scientific 
review committees approved the study. 

Randomization and masking 

Before randomization, meetings were held with the man- 
agement of the eight shortlisted hospitals in which the 
study design, mode of data collection, longevity of the 
study and intervention were discussed. Each hospital held 
internal discussion after which the study team sought 
assent regarding the hospital's participation in the study. 
After obtaining assent from all eight hospitals, the hospi- 
tals were allocated to either the full intervention or partial 
intervention using restricted randomization. Hospitals 
coded as H1-H4 received the full intervention while hospi- 
tals H5-H8 received a partial intervention (control). Of 
the 70 possible randomization outcomes to the interven- 
tion and control groups, seven gave relatively balanced 
groups in terms of hospital level covariates, and one of 
these was randomly chosen using a "blind draw" proced- 
ure. It was not possible to mask treatment of participating 
hospitals, but details on group allocations were not pub- 
licly disseminated, geographical distance between hospitals 
was relatively large and there is typically little formal effort 
to transfer knowledge and practice between hospitals. 

Study intervention 

The intervention was delivered over an 18 month-period 
(from September 2006 to April 2008) and aimed to im- 
prove quality of pediatric admission care through imple- 
mentation of best practice guidelines and local efforts to 
tackle organizational constraints. The partial intervention 
delivered in control hospitals involved a 1.5 day lecture- 
based introductory seminar explaining the evidence based 
clinical practice guidelines followed by dissemination of 
these guidelines and accompanying job aides, and regular 
hospital performance assessment surveys conducted every 
6 months followed by written feedback. Conversely, the 
intervention hospitals received 5.5 day training on ETAT + 
[2], a local hospital facilitator responsible for on-site prob- 
lem solving who received external supervisory support by 
telephone from the implementation team, and face to face 
feedback of survey findings at the end of each survey (see 
[13] for full description). The package delivered in inter- 
vention hospitals was in addition to written feedback and 
clinical guideline and job aide dissemination. 

Data collection 

Data relevant to this study were collected at baseline, six 
months post baseline and at the end of intervention (18 
months) in both control and intervention hospitals. Data 
collection teams received three-weeks training including 
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a pilot survey prior to baseline data collection with fur- 
ther details of procedures supplied elsewhere [13]. At 
baseline and for each subsequent round up to four data 
collection teams working concurrently spent two weeks 
at each hospital collecting retrospective data from a ran- 
dom sample of medical records of children discharged 
over the preceding six months [6]. During each survey 
one team member was assigned to collect prospective 
data on process of care for all children present on the 
wards at the start of the survey and every child admitted 
during the two weeks period that followed. Therefore 
the retrospective dataset covers children admitted during 
the six months period preceding the survey while the 
prospective data set covers children admitted during the 
two weeks period that data collection was taking place. 
The aim was to enroll 50 cases per survey per hospital 
based on estimated admission rates for the size of 
hospital studied. Data were abstracted on standardized 
forms from medical records and other supporting docu- 
ments such as nursing charts and laboratory requests 
with clarification being sought form health workers or 
children's caretakers as needed. For quality assurance 
purposes team leaders assessed data quality for all cases 
and independently re-evaluated a 10% sample of retro- 
spective and prospective case records during data col- 
lection. Ethical approval was granted for confidential 
abstraction of data from case records without individ- 
uals' consent. 



Performance indicators 

The primary outcome was change in quality of pediatric 
care measured using 13 process of care indicators in inter- 
vention versus control hospitals. These process indicators, 
the same as those used for the retrospective data analysis 
[6], were derived from evidence based clinical guideline 
recommendations for management of pneumonia, malaria 
and diarrhea and/or dehydration. Three additional indi- 
cators focusing on key policy recommendations for paedi- 
atric care were vitamin A administration on admission, 
provider initiated HIV testing and identification of missed 
opportunities for vaccination. Mortality was not a primary 
outcome. 

All the 13 process indicators were dichotomous vari- 
ables indicating whether the patient assessment, treat- 
ment and supportive care were implemented according 
to guidelines. An overall score for assessment was calcu- 
lated representing the proportion of relevant assessment 
tasks completed for each child. This score, constrained 
between zero and one, was derived from a maximum 
possible number of assessment indicators for each child, 
with 5 indicators for all children, and extra indicators for 
children diagnosed with malaria (4), pneumonia (4) and 
diarrhea/ dehydration (2) (Additional file 1). 



Sample size 

The data collection period for each hospital in each survey 
was restricted to two weeks limiting the number of pro- 
spectively observed patient episodes that could be col- 
lected to approximately 50 cases per hospital per survey 
given the workload in the selected hospitals. Since the 
number of prospective observations was limited, we ex- 
plored the ability to detect important effect sizes using typ- 
ical values of power of 80%, 95% precision and with only 4 
clusters per arm with intra-class correlation coefficients 
derived from retrospective data analysis (Additional file 2). 
For example, assuming 25 malaria cases per site at the 18 
months survey and 50% correct management in control 
hospitals, the differences between intervention and control 
arms that would seem an unlikely chance finding ranged 
from greater than 9% to greater than 20% for ICC values 
of 0.008 and 0.226 respectively (see Additional file 2). 
Pooling data across the two post intervention surveys 
would allow identification of smaller apparent differ- 
ences (but taking no account of multiple comparisons). 

Statistical analysis 

Data from the retrospective study were a sub-set of the 
data from four surveys (including one at 12 months) used 
by Ayieko et al. [6]. For the prospective study, data entry 
was conducted independently by two clerks using Micro- 
soft Access databases and verified. For both studies (pro- 
spective and retrospective) data analyses were conducted 
using Stata version 11 [14]. Descriptive sample character- 
istics were calculated at the hospital level for each survey 
period, using medians, and inter-quartile range (IQR) for 
skewed continuous variables and means or proportions 
with 95% confidence intervals (95% CI) for continuous 
and binary categorical variables respectively. 

The effect of the intervention was assessed in 2 ways for 
both the retrospective data and the prospective data. The 
first set of analyses combined data from surveys 2 and 4 
(post intervention) for each hospital, and for each process 
indicator used an unpaired t-test to compare summary 
measures in each hospital (4 intervention and 4 control), 
in order to assess the effect of the intervention [15]. Sum- 
mary measures of each indicator were used to obtain un- 
adjusted risk differences and risk ratios for the effect of 
the intervention. The adjusted risk ratios and risk differ- 
ences were calculated using covariate adjusted cluster re- 
siduals, whereby a logistic regression model is fitted using 
only personal (child-level) factors, and residuals computed 
for each of the 8 hospitals by subtracting observed hospital 
means from predicted hospital means [15]. These hospital 
residuals were compared using an unpaired t-test across 
the 4 intervention and 4 control hospitals. 

A second analysis was undertaken using a multi-level 
logistic regression model for each process indicator, tak- 
ing into account the clustering at the hospital level. The 
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multi-level model used data from all three surveys, not 
adjusting for any individual level characteristics, or hos- 
pital level characteristics, to obtain crude odds ratios 
(OR) to assess the impact of the intervention as the 
interaction between intervention hospitals in survey 2 
and 4 pooled together compared to the main effects of 
the intervention at baseline. This was reported as ratios 
of odds ratios for dichotomous indicators and differ- 
ences in differences for assessment score. A bar chart 
comparing adjusted risk ratios for prospective and retro- 
spective datasets was plotted to provide a visual indica- 
tion of how well the results from the two datasets agree. 

Results 

All eight participating hospitals completed the study 
providing 1295 admission episodes followed up pro- 
spectively. These included 505 episodes from the pre- 
intervention period and 790 from the post intervention 
period (413 at 6 months and 377 at 18 months). A total 
of 6302 retrospective case records were available for 
analysis from similar surveys. The characteristics of chil- 
dren whose admissions were evaluated prospectively 
were similar to those in the retrospective dataset, Table 1. 
Full description of process indicators for both prospect- 
ive and retrospectively datasets provided in Additional 
files 3 and 4 respectively show improvement for many of 
those indicators between baseline and follow up at 6 
months and 18 months. No hospital received additional 
training from external sources or other intervention 
components during the study period. 

The adjusted risk difference and risk ratio between inter- 
vention and control hospitals for the process indicators 



obtained from the pooled (6 months plus 18 months) pro- 
spective data and for the retrospective data pooled over 
the same periods are shown in Table 2 (unadjusted results 
are shown in Table 3). The magnitude of the intervention 
effect varied across the process indicators. For the pro- 
spective data there was strong evidence of an intervention 
effect for 8 of the indicators and weaker evidence of an ef- 
fect for one indicator, with magnitude of effect sizes vary- 
ing from 23% to 60% difference. For the retrospective 
data, 10 process indicators had statistically significant dif- 
ferences with magnitude of effects varying from 10% to 
42%. The mean risk difference between intervention and 
control hospitals using data collected at six months and 
18 months surveys only in both the prospective and retro- 
spective datasets are displayed graphically in Figure 1. This 
illustrates their similarity in terms of magnitude of effects 
and statistical significance for all except two indicators, 
the proportion of patients receiving quinine that received 
an overdose and the proportion of patients receiving 
gentamicin that received an under dose. In both of these 
indicators, the retrospective dataset gave more favorable 
results. These discrepancies were possibly caused by the 
small number of patients who had either a gentamicin or 
quinine prescription on which to calculate these dosage 
error indicators particularly in the prospective data set. 

The results from multilevel models comparing the effect 
of intervention during the post-intervention period to that 
at baseline for both prospective and retrospective datasets 
are presented in Table 4. The prospective data showed 
statistically significant differences between baseline and 
post-intervention periods for 8 process indicators. These 
differences were also observed in the retrospective dataset 



Table 1 Demographic characteristic of children by survey during prospective and retrospective data collection 

Control Intervention 



Prospective Retrospective Prospective Retrospective 



Baseline (Survey 1) 

Number of admission episodes included in analysis 253 1005 252 1130 

Mean (SD) age (months) 16.2(13.3) 15.8(12.5) 17.0(13.3) 15.8(12.7) 

Number (%) Male 101(56.1) 389(55.0) 72(53.3) 241(53.4) 

Number (%) of children with structured admission form in medical records 24.0(9.5) 30.0(3.0) 0.0(0.0) 0.0(0.0) 
6 month (survey 2) 

Number of admission episodes included in analysis 215 840 198 1022 

Mean (SD) age (months) 14.4(12.3) 16.7(13.4) 14.6(12.1) 17.1(13.1) 

Number (%) Male 97(50.5) 406(54.1) 88(56.4) 487(56.3) 

Number (%) of children with structured admission form in medical records 98.0(46.2) 590.0(70.9) 185.0(96.4) 929.0(91.2) 
18 months (survey 4) 

Number of admission episodes included in analysis 178 1147 199 1158 

Mean (SD) age (months) 17.4(14.3) 15.7(12.8) 18.2(14.9) 16.2(12.8) 

Number (%) Male 93(58.9) 588(53.0) 92(51.1) 639(58.0) 

Number (%) of children with structured admission form in medical records 80.0(45.7) 642.0(56.0) 178.0(90.8) 1 1 12.0(96.2) 



Table 2 Adjusted differences in process of care indicators during the post intervention period 

Prospective Retrospective 
Indicator of quality of care Intervention % Control % Difference [95% CI] Risk ratio [95% CI] Intervention % Control % Difference [95% CI] Risk ratio [95% CI] 

All Children 



Childs weight documented 


79.1 


45 


36.2[21 .0-51.4] 


1 .911 .4-2.6] 


82.8 


59.1 


26.8[1 2.8-40.8] 


1.5[1 .2-1.9] 


Child's temperature documented 


63 


30.5 


30.2[-1 0.3-70.7] 


3.7[0.6-22.4] 


64.9 


46 


1 9.4[- 14.8-53.7] 


1 .9[0.6-5.8] 


Vitamin A Administered on Admission 


16.4 


17.3 


-0.3 [-8.5-7.9] 


1 ,2[0.7-2.0] 


22.2 


11.9 


10.5 [-5.3-26.3] 


2.1 [0.5-9.9] 


Provider Initiated HIV testing 


18.1 


15.8 


3.3[-1 0.3-1 6.9] 


1.3[0.5-3.2] 


16 


3.5 


13.0[1 0.1-15.9] 


5.8[2.6-12.7] 


Vaccination status documented 


41.3 


18.6 


23.9[6.0-41.7] 


3.1 [0.9-1 0.0] 


50.8 


27.2 


26.9[9.9-44.0] 


2.6[1 .2-5.6] 


Average assessment score (range, 0-1) 


92.1 


58.7 


50.1 [23.9-76.3] 




92.6 


67.2 


49.9[30.5-69.2] 




Children with diagnosis of malaria 


















Proportion of malaria with a severity classification 


91.9 


43 


50.9[23.7-78.0] 


2.4[1 .3-4.4] 


89.1 


46.8 


41. 4[1 4.3-68.5] 


2.0[1 .1-3.5] 


Proportion with quinine loading dose 


88.5 


59.3 


46.4[5.9-86.8] 


2.5[0.9-7.1] 


86.9 


58.1 


37.6[9.0-66.2] 


2.0[1 .0-3.8] 


Proportion with twice daily quinine maintenance dose 


94.1 


74.9 


24.7[1 1.4-38.1] 


1.4[1.1-1.7] 


79 


42.5 


37.5[1 4.2-60.7] 


1.9[1. 3-2.8] 


Proportion with quinine daily dose > =40 mg/kg 


4 


I 


3.0[-1. 4-7.4] 




2.2 


8 


-7.0[-1 3.4-0.5] 


0.2 [0.1 -0.6] 


Children with diagnosis of pneumonia 


















Proportion of pneumonia with a severity classification 


95.8 


63.6 


46.8[-1. 2-94.8] 


1.5[1. 1-2.0] 


93.3 


70.1 


33.5[-2.7-69.7] 


1.7[0.9-3.1] 


Proportion with once daily gentamicin dose 


86.8 


57.6 


27.6[1 1.6-43.5] 


1.5[1. 1-2.1] 


79.1 


68.1 


14.9[9.7-20.1] 


1 ,2[1.1-1.4] 


Proportion with gentamicin daily dose <4 mg/kg 


3.9 


3.4 


2.3[-3.3-7.9] 


1 ,9[0.8-4.3] 


3.9 


15.3 


-10.2[-1 2.3-8.1] 


0.3 [0.2-0.4] 


Proportion with gentamicin daily dose > =10 mg/kg 


4.7 


6.8 


-0.4[-6.0-5.2] 


1 .0(0.5-2.2] 


8.6 


9.9 


-03[-3.1-2.5] 


1.0[0.7-1.4] 


Children with diagnosis of diarrhea/dehydration 


















Proportion of diarrhoea/dehydration diagnosis with 
a severity classification 


98.6 


85.2 


8.2 [-16.3-32.8] 


1.1 [0.8- 1.5] 


97.9 


89.1 


14.7[2.9-26.5] 


1.2D.0-1.4] 


Correct fluid prescription 


67.3 


11.5 


60.2[23.5-96.9] 


2.7[1 .4-5.3] 


62.2 


32.7 


33.0[22.8-43.2] 


2.1 [1.6-2.9] 



Table 3 Unadjusted differences in process of care indicators during the post intervention period 

Prospective Retrospective 
Indicator of quality of care Intervention % Control % Difference [95% CI] Risk ratio [95% CI] Intervention Control Difference [95% CI] Risk ratio [95% CI] 

All Children 



Childs weight documented 


79.1 


45 


34.1 [27.7-40.4] 


1 .8[1 .6-2.0] 


82.8 


59.1 


23.7[21. 0-26.4] 


1.4[1 .3-1.5] 


Child's temperature documented 


63 


30.5 


32.4[25.9-39.0] 


2.1 [1.7-2.4] 


64.9 


46 


18.9[1 5.9-2 1.8] 


1.4[1 .3-1.5] 


Vitamin A Administered on Admission 


16.4 


17.3 


-0.9[-6.1-4.3] 


0.9[0.7-1.3] 


22.2 


11.9 


10.318.0-12.5] 


1.9[1.6-2.1] 


Provider Initiated HIV testing 


18.1 


15.8 


2.3[-2.9-7.6] 


1.1 [0.8-1.6] 


16 


3.5 


12.6D 0.8-1 4.3] 


4.6[3.6-6.0] 


Vaccination status documented 


41.3 


18.6 


22.7[1 6.6-28.9] 


2.2 [1.8-2.8] 


50.8 


27.2 


23.6[20.8-26.5] 


1 .9[1 .7-2.0] 


Average assessment score (range, 0-1) 


92.1 


58.7 


33.4[29.9-36.9] 




92.6 


67.2 


25.4[23.9-26.9] 




Children with diagnosis of malaria 


















Proportion of malaria with a severity classification 


91.9 


43 


48.9[42. 1-55.7] 


2.1 [1.9-2.5] 


89.1 


46.8 


42.3[39.1-45.5] 


1 ,9[1 .8-2.0] 


Proportion with quinine loading dose 


88.5 


59.3 


29.2[20.5-37.9] 


1.5[1.3-1.7] 


86.9 


58.1 


28.8[24.8-32.8] 


1 j5[1 .4-1.6] 


Proportion with twice daily quinine maintenance dose 


94.1 


74.9 


19.2[1 2.2-26.2] 


1.3[1.1-1.4] 


79 


42.5 


36.5[32.0-41.1] 


1.9[1. 7-2.0] 


Proportion with quinine daily dose > =40 mg/kg 


4 


1 


2.9[-0.1-6.0] 


3.8[0.8-1 7.6] 


2.2 


8 


-5.8[-8.0-3.6] 


0.3 [0.2-0.4] 


Children with diagnosis of pneumonia 


















Proportion of pneumonia with a severity classification 


95.8 


63.6 


32.2[24.0-40.4] 


1.5D.3-1.7] 


93.3 


70.1 


23.2[1 9.9-26.6] 


1.3[1.3-1.4] 


Proportion with once daily gentamicin dose 


86.8 


57.6 


29.2[1 9.9-38.5] 


1.5D.3-1.7] 


79.1 


68.1 


11.1 [6.7-1 5.4] 


1.2[1. 1-1.2] 


Proportion with gentamicin daily dose <4 mg/kg 


3.9 


3.4 


0.5[-3.8-4.8] 


1.1 [0.4-3.7] 


3.9 


15.3 


-11.4[-14.3-8.6] 


0.3 [0.2-0.4] 


Proportion with gentamicin daily dose > =10 mg/kg 


4.7 


6.8 


-2.1 [-7.3-3.1] 


0.7[0.3-1.8] 


8.6 


9.9 


-1.2[-4.1-1.6] 


0.9[0.6-1.2] 


Children with diagnosis of diarrhea/dehydration 


















Proportion of diarrhoea/dehydration diagnosis with 
a severity classification 


98.6 


85.2 


1 3.4[3.6-23.3] 


1.2D.0-1.3] 


97.9 


89.1 


8.7[5.9-1 1.6] 


1.U1.1-1.1] 


Correct fluid prescription 


67.3 


11.5 


55.8[37.8-73.8] 


5.8[2.0-17.2] 


62.2 


32.7 


29.5[22.9-36.1] 


1 ,9[1 .6-2.2] 
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8-Proportion with quinine loading dose 16-Correct fluid prescription 

7-Proportion of malaria with a severity classification 15-Proportion of diarrhoea/ dehydration diagnosis with a severity classification 

6-Average assessment score(range, 0-1 ) 14-Proportion with gentamicin daily dose >=10mg/kg 

5-Vaccination status documented 13-Proportion with gentamicin daily dose <4mg/kg 

4-Provider Initiated HIV testing 12-Proportion with once daily gentamicin dose 

3-Vitamin A Administered on Admission 1 1 -Proportion of pneumonia with a severity classification 

2-Child's temperature documented 10-Proportion with quinine daily dose >=40mg/kg 

1 -Childs weight documented 9-Proportion with twice daily quinine maintenance dose 



Figure 1 Comparison of prospective and retrospective risk differences. 



except for HIV testing that showed favorable results in 
retrospective data. It was not possible to calculate ratio of 
odds ratios for one indicator (correct fluid prescription for 
patients presenting with diarrhoea and/or dehydration) in 
the prospective dataset due to a small number of observa- 
tions. Analysis of retrospective data suggested a favorable 
effect of the intervention for this indicator. 

Discussion 

We evaluated a multi-faceted approach to implemen- 
tation of clinical guidelines aimed at treatment of 
illnesses that cause most deaths in Kenyan district hos- 
pitals. We used data collected by personnel present in 
the hospital for two week survey periods prior to inter- 
vention and 6 and 18 months after intervention. The 
data showed that prior use of guideline recommended 
practices for treatment of children with severe illness 
was poor at baseline. Data further showed marked im- 
provement in adoption of guideline recommended prac- 
tices in both partial and full intervention groups but 
improvements were more marked in the full intervention 
group. It is worth noting that improvements were sus- 
tained between 6 to 18 months after initial training despite 
very high staff turn-over amongst the junior clinicians re- 
sponsible for much care. Indeed, of 109 clinical staff re- 
sponsible for attending to the patients sampled as part of 
the retrospective data analysis in survey 4 in the interven- 
tion hospitals, only nine (8.3%) had received any specific 
formal or ad hoc guideline-related training [6]. 



Retrospective data are more efficiently collected than 
prospective data, and can cover a wider time period redu- 
cing possible effects of seasonal or other temporal varia- 
tions. However, problems of missing data may be greater 
when using retrospective case record data, a data quality 
problem potentially overcome by using prospective collec- 
tion. As demonstrated in Figure 1, the analysis of prospect- 
ively collected data gave similar results to those obtained 
when using retrospective data. This consistency helps 
validate, through triangulation, the methods used for 
retrospectively collecting case record data in assessing 
the quality of care. The results provide reassurance that 
assessing practice performance using retrospective sam- 
pling is of value. 

The underlying hypothesis behind these analyses is 
that the assessment of quality of care will be similar in 
retrospective and prospective patients. The analysis rec- 
ommended by Hayes is robust and simple, which makes 
it ideal for hospital surveys in countries with limited re- 
sources. In this paper we had individual level data 
allowing the analysis to adjust for confounders that may 
be associated with the quality of care received by these 
children. The analysis shows the results from the retro- 
spective and prospective patients gave similar results 
for the impact of the intervention on quality of care. In 
the multi-level the assessment of the impact of the 
intervention is made through an interaction effect be- 
tween survey time (6 & 18 months vs. baseline at 0 
months) and the intervention. While this may have the- 
oretical advantages, it may be more sensitive to small 



Table 4 Effect of training for baseline and post-intervention period in prospective data and retrospective data 



Prospective 



Retrospective 



Indicator 

All Children 

Childs weight documented 
Child's temperature documented 
Vitamin A Administered on Admission 
Provider Initiated HIV testing 
Vaccination status documented 
Average assessment score (range, 0-1) 

Children with diagnosis of malaria 

Proportion of malaria with a severity 
classification 

Proportion with quinine loading dose 

Proportion with twice daily quinine 
maintenance dose 

Proportion with quinine daily 
dose > =40 mg/kg 

Children with diagnosis of pneumonia 

Proportion of pneumonia with a severity 
classification 

Proportion with once daily gentamicin dose 

Proportion with gentamicin daily dose 
<4 mg/kg 

Proportion with gentamicin daily 
dose > =10 mg/kg 

Children with diagnosis of diarrhea/ 
dehydration 

Proportion of diarrhoea/dehydration 
diagnosis 

with a severity classification 
Correct fluid prescription 



Baseline OR 

3.95(1.92-8.15) 
0.68(0.07-6.23) 
0.23(0.06-0.93) 
13.08(2.63-65.14) 
0.08(0.02-0.38) 
Mean 



Post intervention OR 

4.99(2.51-9.94) 
4.09(0.46-36.08) 
1.18(0.40-3.53) 
1.21(0.56-2.61) 
4.56(1.22-17.09) 



Ratio of OR 

1.26(0.76-2.10) 
5.99(3.10-11.57) 
5.08(1.87-13.84) 
0.09(0.02-0.42) 
56.29(22.06-143.58) 



Baseline OR 

9.32(3.28-26.46) 
0.81(0.09-7.34) 
0.32(0.07-1.55) 
2.24(0.97-5.18) 
0.23(0.06-0.81) 



Post intervention OR 

4.06(1.45-11.39) 
1.75(0.20-15.59) 
2.44(0.53-11.29) 
5.65(3.57-8.93) 
4.38(1.24-15.43) 



0.04(-0.1 7-0.09) Mean = 0.33(0.20-045) Differences in Means = 0.37 Mean = 0.01 (-0.1 0-0. 12) Mean = 0.27(0.16-0.38) 

(0.32-0.41) 



2.92(0.95-8.96) 

5.72(0.95-34.50) 
4.37(1.07-17.89) 

2.68(0.16-44.63) 



2.00(0.34-11.71) 

0.65(0.11-3.93) 
2.76(0.60-12.67) 

1 .54(0.28-8.55) 



0.23(0.03-1.59) 



3.9e + 06(0.00- .) 



1 5.28(5.80-40.27) 

7.38(2.22-24.61) 
5.68(2.30-14.03) 

3.25(0.21-49.51) 



23.33(3.93-138.58) 

6.09(1.05-35.31) 
1 .38(0.26-7.48) 

0.67(0.24-1.84) 



12.80(1.42-115.47) 



15.81(4.13-60.59) 



5.22(2.05-13.31) 

1.29(0.28-5.97) 
1.30(0.31-5.48) 

1.21(0.09-16.85) 



1 1 .68(3.40-40.07) 

9.34(3.05-28.57) 
0.50(0.10-2.63) 

0.44(0.06-3.19) 



55.99(3.03-1034.44) 



0.00(0.00- 



6.10(2.35-15.85) 

0.41(0.14-1.19) 
0.05(0.01-0.24) 

1.45(0.77-2.71) 



1 .72(0.59-4.99) 

0.29(0.11-0.76) 
2.20(1.18-4.10) 

0.44(0.24-0.81) 



0.78(0.47-1 .28) 



0.18(0.06-0.59) 



8.80(4.01-19.28) 

6.63(2.47-1 7.76) 
6.45(2.23-18.65) 

0.50(0.24-1.02) 



7.29(2.70-1 9.66) 

1.81(0.95-3.43) 
0.21(0.11-0.41) 

0.90(0.59-1.38) 



5.85(3.07-11.17) 



3.57(1.72-7.42) 



Ratio of OR 

0.44(0.33-0.57) 

2.15(1.53-3.03) 

7.61(4.93-11.75) 

2.52(1.14-5.57) 

19.42(14.07-26.81) 

Differences in Means = 0.26 
(0.24-0.28) 

1 .44(0.77-2.69) 

16.24(9.28-28.41) 
131.29(37.51-459.47) 

0.35(0.17-0.68) 



4.24(2.40-7.48) 

6.24(2.78-14.02) 
0.09(0.05-0.16) 

2.07(1.07-4.00) 



7.53(3.63-15.62) 



1 9.56(7.04-54.40) 
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numbers, especially in the estimation of the interaction, 
and the results were much more varied than with the sim- 
pler analysis. The distributional assumptions of multilevel 
models are difficult to verify when we have few clusters as 
in our case but these models allow for examination of 
intervention effect and effect of other covariates simultan- 
eously which is not possible using the method proposed 
by Hayes. 

Limitations 

The main limitation of the study design was the fact that 
hospitals were not selected at random from a list of all 
eligible Kenyan hospitals. While primarily for logistical 
reasons the ability to include only a few hospitals (clus- 
ters) undermine notions of generalizability and balance 
even if random selection and random allocation are used 
[16]. Masking could clearly not be done and this could 
have resulted in bias during data collection, arguably 
particularly during prospective data collection. We how- 
ever tried to minimize such bias by extensive training in 
survey methods and use of standard operating proce- 
dures. This report also illustrates how we have tried to 
triangulate findings using different datasets likely subject 
to different potential biases. Finally as there were only 
four hospitals in each treatment group, efforts to adjust 
for baseline characteristics may not have been as suc- 
cessful as we would have liked. 

Conclusion 

This study helps strengthen the growing evidence 
[17,18] that multifaceted interventions can help improve 
adoption of guidelines and more generally the quality of 
pediatric care. Carrying out studies of similar interven- 
tion packages to improve paediatric care in other low in- 
come settings would help strengthen the evidence base 
most applicable to these settings and provide opportun- 
ities for understanding the influence of different contexts 
on effectiveness [19]. While earlier reports based on 
retrospectively collected data supported the effectiveness 
of the intervention examined, the analysis of prospect- 
ively collected data described here serves to support 
those findings while overcoming potential bias inherent 
to the use of retrospective case record review. The simi- 
lar inference reached after analyses based on prospective 
assessment of case management is a useful finding as it 
supports the utility of work based on examination of 
retrospectively assembled case records allowing longer 
time periods to be studied while constraining costs. Fur- 
ther we have been able to develop and apply a suite of 
analytical methods for assessing the impact of interven- 
tions aimed at improving paediatric hospital care that 
should inform the design and conduct of future studies 
in this field. 



Additional files 



Additional file 1: Process Indicators with Respective Diagnoses. 

Additional file 2: Minimum Detectable Effect Sizes Given Fixed 
Number of Clusters and Varying Baseline Proportions, ICC, and 
Cluster Sample Sizes. The figure shows minimum detectable differences 
given four clusters per arm and baseline proportions of 0.05, 0.1, 0.3, and 
0.5. The intra class correlation coefficients for the quality indicators as 
estimated from the retrospective data and ranged from 0.08 for 
proportion on pneumonia cases that had a gentamicin overdose to 0.226 
for temperature documentation. This work indicate that for malaria, 
assuming 25 cases per site per survey and 50% correct management in 
control hospitals, the difference between intervention and control arms 
could only be detected if it was greater than 9% and 20% for an ICC of 
0.008 and 0.226 respectively. 

Additional file 3: Process indicators by hospital during each survey, 
from prospective data. 

Additional file 4: Process indicators by hospital during each survey, 
from prospective data. 
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